Immersive viewing experience

ABSTRACT

This patent discloses a method record imagery in a way that is larger than a user could visualize. Then, allow the user to view naturally via head tracking and eye tracking to allow one to see and inspect a scene as if one were naturally there viewing it in real time. A smart system of analyzing the viewing parameters of a user and streaming of customized image to be displayed is also taught herein.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in part of U.S. patent applicationSer. No. 17/225,610 filed on 7 Apr. 2021, which is acontinuation-in-part of U.S. patent application Ser. No. 17/187,828filed on Feb. 28, 2021.

TECHNICAL FIELD

Aspects of this disclosure are generally related to use of distributionof work.

INTRODUCTION

Movies are a form of entertainment.

SUMMARY

All examples, aspects and features mentioned in this document can becombined in any technically conceivable way. This patent teaches amethod, software and apparatus for an immersive viewing experience.

In general, this patent improves on techniques taught in U.S. patentapplication Ser. No. 17/225,610 filed on 7 Apr. 2021, which isincorporated by reference in its entirety. Some of the apparatusesdescribed in U.S. patent application Ser. No. 17/225,610 havecapabilities to generate extremely large datasets. This patent improvedthe display of such extremely large datasets.

This patent discloses, a system, a method, an apparatus and software toachieve an improved immersive viewing experience. First, upload a user'sviewing parameter to a cloud wherein said cloud stores imagery (which inthe preferred embodiments is extremely large datasets). Viewingparameters can include any action, gesture, body position, eye lookangle, eye convergence/vergence or input (e.g., via a graphical userinterface). Thus, in near real time, user's viewing parameters arecharacterized (e.g., by a variety of devices, such as eye-facingcameras, cameras to record gestures) and sent to the cloud. Second, aset of user-specific imagery is optimized from said imagery wherein saiduser-specific imagery is based on at least said viewing parameter. Inthe preferred embodiment, the field of view of the user-specific imageryis smaller than the imagery. In the preferred embodiment, the locationwhere a user is looking would have high resolution and the locationwhere the user is not looking would have low resolution. For example, ifa user is looking at an object on the left, then the user-specificimagery would be high resolution on the left side. In some embodiments,a user-specific imagery would be streamed in near-real time.

In some embodiments, the user-specific imagery comprises a first portionwith a first spatial resolution and a second portion with a secondspatial resolution, and wherein said first spatial resolution is higherthan said second spatial resolution. Some embodiments comprise whereinsaid viewing parameter comprises a viewing location and wherein saidviewing location corresponds to said first portion.

Some embodiments comprise wherein user-specific imagery comprises afirst portion with a first zoom setting and a second portion with asecond zoom setting, and wherein said first zoom setting is higher thansaid second zoom setting. Some embodiments comprise wherein a firstportion is determined by said viewing parameter wherein said viewingparameter comprises at least one of the group consisting of: a positionof said user's body; an orientation of said user's body; a gesture ofsaid user's hand; a facial expression of said user; a position of saiduser's head; and an orientation of said user's head. Some embodimentscomprise wherein a first portion is determined by a graphical userinterface, such as a mouse or controller.

Some embodiments comprise wherein the imagery comprises a first field ofview (FOV) and wherein said user-specific imagery comprises a secondfield of view, and wherein said first FOV is larger than said secondFOV.

Some embodiments comprise wherein imagery comprises stereoscopic imageryand wherein said stereoscopic imagery is obtained via stereoscopiccameras or stereoscopic camera clusters.

Some embodiments comprise wherein said imagery comprises stitchedimagery wherein said stitched imagery is generated by at least twocameras.

Some embodiments comprise wherein said imagery comprises compositeimagery, wherein said composite imagery is generated by: taking an firstimage of a scene with a first set of camera settings wherein said firstset of camera settings causes a first object to be in focus and a secondobject to be out of focus; and taking an second image of a scene with asecond set of camera settings wherein said second set of camera settingscauses said second object to be in focus and said first object to be outof focus. Some embodiments comprise wherein when user looks at saidfirst object, said first image would be presented to said user and whenuser looks at said second object, said second image would be presentedto said user. Some embodiments comprise combining at least said firstobject from said first image and said second object from said secondimage into said composite image.

Some embodiments comprise wherein image stabilization is performed. Someembodiments comprise wherein said viewing parameter comprisesconvergence. Some embodiments comprise wherein user-specific imagery is3D imagery wherein said 3D imagery is presented on a HDU, a set ofanaglyph glasses or a set of polarized glasses.

Some embodiments comprise wherein said user-specific imagery ispresented to said user on a display wherein said user has at least a0.5π steradian field of view.

Some embodiments comprise wherein user-specific imagery is presented ona display. In some embodiments, the display is a screen (e.g., TV,reflective screen coupled with a projector system, an extended realityhead display unit including an augmented reality display, a virtualreality display or a mixed reality display).

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates the FIG. 1 illustrates retrospective display ofstereoscopic images.

FIG. 2 illustrates methods to determine which stereo pair to display toa user for a given time point.

FIG. 3 illustrates displaying a video recording on a HDU.

FIG. 4 illustrates a pre-recorded stereo viewing performed by user 1.

FIG. 5 illustrates performing long range stereoscopic imaging of adistant object using stereoscopic camera clusters.

FIG. 6 illustrates a capability of post-acquisition adjusting the imagesto bring into the best possible picture based on user eye tracking bythe generation of a stereoscopic composite image.

FIG. 7A illustrates an image with motion and the application of imagestabilization processing.

FIG. 7B illustrates an image with motion displayed in a HDU.

FIG. 7C illustrates an image stabilization applied to the image usingstereoscopic imagery.

FIG. 8A illustrates a left image and a right image with a first camerasetting.

FIG. 8B illustrates a left image and a right image with a second camerasetting.

FIG. 9A illustrates a top down view of all data gathered of a scene at atime point.

FIG. 9B illustrates a displayed wide angle 2D image frame of the videorecording.

FIG. 9C illustrates a top down view of User A's viewing angle of −70°and 55° FOV.

FIG. 9D illustrates what User A would see given User A's viewing angleof −70° and 55° FOV.

FIG. 9E illustrates a top down view of User B's viewing angle of +50°and 85° FOV.

FIG. 9F illustrates what User B would see given User B's viewing angleof +50° and 85° FOV.

FIG. 10A illustrates the field of view captured at a first time point bythe left camera.

FIG. 10B illustrates the field of view captured at a first time point bythe right camera.

FIG. 10C illustrates a first user's personalized field of view (FOV) ata given time point.

FIG. 10D illustrates a second user's personalized field of view (FOV) ata given time point.

FIG. 10E illustrates a third user's personalized field of view (FOV) ata given time point.

FIG. 10F illustrates a fourth user's personalized field of view (FOV) ata given time point.

FIG. 11A illustrates a top down view of the first user's left eye view.

FIG. 11B illustrates a top down view of the first user's left eye viewwherein a convergence point in close proximity to the left eye and righteye.

FIG. 11C illustrates a left eye view at time point 1 withoutconvergence.

FIG. 11D illustrates a left eye view at time point 2 with convergence.

FIG. 12 illustrates the reconstruction of various stereoscopic imagesfrom previously acquired wide angle stereo images.

FIG. 13A illustrates a top down view of a home theater.

FIG. 13B illustrates a side view of the home theater as shown in FIG.13A.

FIG. 14A illustrates a top down view of a home theater.

FIG. 14B illustrates a side view of the home theater as shown in FIG.14A.

FIG. 15A illustrates a near-spherical TV with a user looking straightahead at time point #1.

FIG. 15B shows the portion of the TV and the field of view beingobserved by the user at time point #1.

FIG. 15C illustrates a near-spherical TV with a user looking straightahead at time point #2.

FIG. 15D shows the portion of the TV and the field of view beingobserved by the user at time point #2.

FIG. 15E illustrates a near-spherical TV with a user looking straightahead at time point #3.

FIG. 15F shows the portion of the TV and the field of view beingobserved by the user at time point #3.

FIG. 16A illustrates an un-zoomed image.

FIG. 16B illustrates a digital-type zooming in on a portion of an image.

FIG. 17A illustrates an un-zoomed image.

FIG. 17B illustrates the optical-type zooming in on a portion of animage.

FIG. 18A illustrates a single resolution image.

FIG. 18B illustrates a multi-resolution image.

FIG. 19A illustrates a large field of view wherein a first user islooking at a first portion of the image and a second user is looking ata second portion of the image.

FIG. 19B illustrates that only the first portion of the image in FIG.19A and that the second portion of the image in FIG. 19A are highresolution and the remainder of the image is lower resolution.

FIG. 20A illustrates a low resolution image.

FIG. 20B illustrates a high resolution image.

FIG. 20C illustrates a composite image.

FIG. 21 illustrates a method and a process for performing near-real-timestreaming of customized images.

FIG. 22A illustrates using resection in conjunction with stereoscopiccameras wherein a first camera location is unknown.

FIG. 22B illustrates using resection in conjunction with stereoscopiccameras wherein an object location is unknown.

FIG. 23A illustrates a top down view of a person looking forward to thecenter of the screen of the home theater.

FIG. 23B illustrates a top down view of a person looking forward to theright side of the screen of the home theater.

FIG. 24 illustrates a method, system and apparatus for optimizingstereoscopic camera settings during image acquisition during movement.

DETAILED DESCRIPTIONS

The flow diagrams do not depict the syntax of any particular programminglanguage. Rather, the flow diagrams illustrate the functionalinformation one of ordinary skill in the art requires to fabricatecircuits or to generate computer software to perform the processingrequired in accordance with the present invention. It should be notedthat many routine program elements, such as initialization of loops andvariables and the use of temporary variables, are not shown. It will beappreciated by those of ordinary skill in the art that unless otherwiseindicated herein, the particular sequence of steps described isillustrative only and can be varied without departing from the spirit ofthe invention. Thus, unless otherwise stated the steps described beloware unordered meaning that, when possible, the steps can be performed inany convenient or desirable order.

FIG. 1 illustrates retrospective display of stereoscopic images. 100illustrates step A, which is to determine a location (e.g., (α_(n),β_(n), r_(n)) coordinate) where a viewer is looking at time point n.Note #1: This location could be a near, medium or far convergence point.Note #2: A collection of stereoscopic imagery has been collected andrecorded. Step A follows the collection process and takes place at somesubsequent time period during viewing by a user. 101 illustrates step B,which is to determine a FOV_(n) corresponding to the location (e.g.,(α_(n), β_(n), r_(n)) coordinate for time point n. Note: user had optionto select the FOV). 102 illustrates step C, which is to select camera(s)that correspond to the FOV for left eye with option to performadditional image processing (e.g., use composite image, use vergencezone) to generate personalized left eye image at time point n(PLEI_(n)). 103 illustrates step D, which is to select camera(s) thatcorrespond to the FOV for right eye with option to perform additionalimage processing (e.g., use composite image, use vergence zone) togenerate personalized right eye image at time point n (PREI_(n)). 104illustrates step E, which is to display PLEI_(n) on a left eye displayof a HDU. 105 illustrates step F, which is to display PREI_(n) on aright eye display of a HDU. 106 illustrates step G, which is toincrement time step to n+1 and go to Step A, above.

FIG. 2 illustrates methods to determine which stereo pair to display toa user for a given time point. 200 illustrates a text box of analyzingthe user's parameters to determine which stereoscopic image to displayto the user. First, use the viewing direction of a user's head. Forexample, if user's head is in a forward direction, a first stereo paircould be used and if a user's head is in a direction toward the left asecond stereo pair could be used. Second, use the viewing angle of theuser's gaze. For example, if user is looking in a direction towards adistant object (e.g., mountain in the distance), then the distant (e.g.,zone 3) stereo image pair would be selected for that time point. Third,use the user's convergence. For example, if a viewing direction of anear object (e.g., leaf on a tree) is extremely similar to a viewingdirection to a distant object (e.g., mountain in the distance), optionto use combination of convergence and viewing angle. Fourth, use theaccommodation of the user's eyes. For example, monitor a user's pupilsize and use change in size to indicate where (near/far) the user islooking.

FIG. 3 illustrates displaying a video recording on a HDU. 300illustrates establishing a coordinate system. For example, use cameracoordinate as the origin and use pointing direction of camera as anaxis. This is discussed in more detail in U.S. patent application Ser.No. 17/225,610, which is incorporated by reference in its entirety. 301illustrates performing wide angle recording of a scene. For example,record data with a FOV larger than the FOV shown to a user). 302illustrates performing an analysis of a user, as discussed in FIG. 2 todetermine where the user is looking at in the scene. 303 illustratesoptimizing the display based on the analysis in 302. In someembodiments, a feature (e.g., position, size, shape, orientation, color,brightness, texture, classification by AI algorithm) of a physicalobject determines a feature (e.g., position, size, shape, orientation,color, brightness, texture) of a virtual object. For example, a user isusing a mixed reality display in a room in a house wherein some of theareas in the room (e.g., a window during the daytime) are bright andsome of the areas in the room are dark (e.g., a dark blue wall). In someembodiments, the position of placement of virtual objects is based onthe location of objects within the room. For example, a virtual objectcould be colored white if the background is a dark blue wall, so that itstands out. For example, a virtual object could be colored blue if thebackground is a white wall, so that it stands out. For example, anvirtual object could be positioned (or re-positioned) so its backgroundis such that the virtual object can be displayed in a fashion tooptimize viewing experience for a user.

FIG. 4 illustrates a pre-recorded stereo viewing performed by user 1.400 illustrates user 1 performing a stereo recording using a stereocamera system (e.g., smart phone, etc.). This is discussed in moredetail in U.S. patent application Ser. No. 17/225,610, which isincorporated by reference in its entirety. 401 illustrates the stereorecording being stored on a memory device. 402 illustrates a user (e.g.,User 1 or other user(s)) retrieving the stored stereo recording. Notethat the stereo recording may be transmitted to the other user(s) andthe other user(s) would receive the stored stereo recording. 403illustrates a user (e.g., User 1 or other user(s)) viewing the storedstereo recording on a stereo display unit (e.g., augmented reality,mixed reality, virtual reality display).

FIG. 5 illustrates performing long range stereoscopic imaging of adistant object using stereoscopic camera clusters. 500 illustratespositioning two camera clusters at at least 50 feet apart. 501illustrates elects a target at at least 1 mile away. 502 illustratesprecisely aiming each camera cluster such that the centerline of focusintersects at the target. 503 illustrates acquiring stereoscopic imageryof the target. 504 illustrates viewing and/or analyzing the acquiredstereoscopic imagery. Some embodiments use cameras with telephoto lensesrather than camera clusters. Also, some embodiments, have stereoseparation of less than or equal to 50 feet apart for optimized viewingof less than 1 mile away.

FIG. 6 illustrates a capability of post-acquisition adjusting the imagesto bring into the best possible picture based on user eye tracking bythe generation of a stereoscopic composite image. The stereoscopicimages displayed at this time point has several objects that might be ofinterest to a person observing the scene. Thus, at each time point, astereoscopic composite image will be generated to match at least oneuser's input. For example, if a user is viewing (eye tracking determinesviewing location) the mountains 600 or cloud 601 at a first time point,then the stereoscopic composite image pair delivered to a HDU would begenerated such that the distant objects of the mountains 600 or cloud601 were in focus and the nearby objects including the deer 603 and theflower 602 were out of focus. If the user was viewing (eye trackingdetermines viewing location) a deer 603, then the stereoscopic compositeimages presented at this frame would be optimized for medium range.Finally, if a user is viewing (eye tracking determines viewing location)a nearby flower 603, then the stereoscopic composite images would beoptimized for closer range (e.g., implement convergence, and blur outdistant items, such as the deer 603, the mountains 600 and the cloud601). A variety of user inputs could be used to indicate to a softwaresuite how to optimize the stereoscopic composite images. Gestures suchas squint could be used to optimize the stereoscopic composite image formore distant objects. Gestures such as lean forward could be used tozoom in to a distant object. A GUI could also be used to improve theimmersive viewing experience.

FIG. 7A illustrates an image with motion and the application of imagestabilization processing. 700A illustrates a left eye image of an objectwherein there is motion blurring the edges of the object. 701Aillustrates a left eye image of an object wherein image stabilizationprocessing has been applied.

FIG. 7B illustrates an image with motion displayed in a HDU. 702illustrates the HDU. 700A illustrates a left eye image of an objectwherein there is motion blurring the edges of the object. 700Billustrates a right eye image of an object wherein there is motionblurring the edges of the object. 701A illustrates a left eye display,which is aligned with a left eye of a user. 701B illustrates a right eyedisplay, which is aligned with a right eye of a user.

FIG. 7C illustrates an image stabilization applied to the image usingstereoscopic imagery. A key task of image processing is the imagestabilization using stereoscopic imagery. 700A illustrates a left eyeimage of an object wherein image stabilization processing has beenapplied. 700B illustrates a left eye image of an object wherein imagestabilization processing has been applied. 701A illustrates a left eyedisplay, which is aligned with a left eye of a user. 701B illustrates aright eye display, which is aligned with a right eye of a user. 702illustrates the HDU.

FIG. 8A illustrates a left image and a right image with a first camerasetting. Note that the text on the monitor is in focus and the distantobject of the knob on the cabinet is out of focus.

FIG. 8B illustrates a left image and a right image with a second camerasetting. Note that the text on the monitor is out of focus and thedistant object of the knob on the cabinet is in focus. A point ofnovelty is using at least two cameras. A first image from a first camerais obtained. A second image from a second camera is obtained. The firstcamera and the second camera are in the same viewing perspectives. Also,they are of the scene (e.g., a still scene or a same time point of anscene with movement/changes). A composite image is generated wherein afirst portion of the composite image is obtained from the first imageand a second portion of the composite image is obtained from the secondimage. Note that in some embodiments, an object within the first imagecan be segmented and the same object within the second image can also besegmented. The first image of the object and the second image of theobject can be compared to see which one has better quality. The imagewith better image quality can be added to the composite image. In someembodiments, however, deliberately selecting some portions to be notclear can be performed.

FIG. 9A illustrates a top down view of all data gathered of a scene at atime point.

FIG. 9B illustrates a displayed wide angle 2D image frame of the videorecording. Note that displaying this whole field of view to a user wouldbe distorted given the mismatch between the user's intrinsic FOV (humaneye FOV) and the camera system FOV.

FIG. 9C illustrates a top down view of User A's viewing angle of −70°and 55° FOV. A key point of novelty is the user's ability to select theportion of the stereoscopic imagery with the viewing angle. Note thatthe selected portion could realistically be up to ˜180°, but not more.

FIG. 9D illustrates what User A would see given User A's viewing angleof −70° and 55° FOV. This improves over the prior art because it allowsdifferent viewers to see different portions of the field of view. Whilea human has a horizontal field of view of slightly more than 180degrees, a human can only read text over approximately 10 degrees of thefield of view, can only assess shape over approximately 30 degrees ofthe field of view and can only assess colors over approximately 60degrees of the field of view. In some embodiments, filtering(subtracting) is performed. A human has a vertical field of view ofapproximately 120 degrees with an upward (above the horizontal) field ofview of 50 degrees and a downward (below the horizontal) field of viewof approximately 70 degrees. Maximum eye rotation however, is limited toapproximately 25 degrees above the horizontal and approximately 30degrees below the horizontal. Typically, the normal line of sight fromthe seated position is approximately 15 degreed below the horizontal.

FIG. 9E illustrates a top down view of User B's viewing angle of +50°and 85° FOV. A key point of novelty is the user's ability to select theportion of the stereoscopic imagery with the viewing angle. Also, notethat the FOV of User B is larger than the FOV of User A. Note that theselected portion could realistically be up to ˜180°, but not morebecause of the limitations of the human eye.

FIG. 9F illustrates what User B would see given User B's viewing angleof +50° and 85° FOV. This improves over the prior art because it allowsdifferent viewers to see different portions of the field of view. Insome embodiments, multiple cameras are recording for a 240° film. In oneembodiment, 4 cameras (each with a 60° sector) for simultaneousrecording. In another embodiment, the sectors are filmedsequentially—one at a time. Some scenes of a film could be filmedsequentially and other scenes could be filed simultaneously. In someembodiments, a camera set up could be used with overlap for imagestitching. Some embodiments comprise using a camera ball systemdescribed in described in U.S. patent application Ser. No. 17/225,610,which is incorporated by reference in its entirety. After the imagery isrecorded, imagery from the cameras are edited to sync the scenes andstitch them together. LIDAR devices can be integrated into the camerasystems for precise camera direction pointing.

FIG. 10A illustrates the field of view captured at a first time point bythe left camera. The left camera 1000 and right camera 1001 are shown.The left FOV 1002 is shown by the white region and is approximately 215°and would have an a ranging from +90° to −135° (sweeping from +90° to−135° in a counterclockwise direction). The area not imaged within theleft FOV 1003 would be approximately 135° and would have an a rangingfrom +90° to −135° (sweeping from +90° to −135° in a clockwisedirection).

FIG. 10B illustrates the field of view captured at a first time point bythe right camera. The left camera 1000 and right camera 1001 are shown.The right FOV 1004 is shown by the white region and is approximately215° and would have an a ranging from +135° to −90° (sweeping from +135°to −90° in a counterclockwise direction). The area not imaged within theright FOV 1005 would be approximately 135° and would have an a rangingfrom +135° to −90° (sweeping from +135° to −90° in a counterclockwisedirection).

FIG. 10C illustrates a first user's personalized field of view (FOV) ata given time point. 1000 illustrates the left camera. 1001 illustratesthe right camera. 1006 a illustrates the left boundary of the left eyeFOV for the first user, which is shown in light gray. 1007 a illustratesthe right side boundary of the left eye FOV for the first user, which isshown in light gray. 1008 a illustrates the left boundary of the righteye FOV for the first user, which is shown in light gray. 1009 aillustrates the right side boundary of the right eye FOV for the firstuser, which is shown in light gray. 1010 a illustrates the center lineof the left eye FOV for the first user. 1011 a illustrates the centerline of the right eye FOV for the first user. Note that the center lineof the left eye FOV 1010 a for the first user and the center line of theright eye FOV 1011 a for the first user are parallel, which isequivalent to a convergence point at infinity. Note that the first useris looking in the forward direction. It is suggested that during filmingof a moving that most of the action in the scene occur in this forwardlooking direction.

FIG. 10D illustrates a second user's personalized field of view (FOV) ata given time point. 1000 illustrates the left camera. 1001 illustratesthe right camera. 1006 b illustrates the left boundary of the left eyeFOV for the second user, which is shown in light gray. 1007 billustrates the right side boundary of the left eye FOV for the seconduser, which is shown in light gray. 1008 b illustrates the left boundaryof the right eye FOV for the second user, which is shown in light gray.1009 b illustrates the right side boundary of the right eye FOV for thesecond user, which is shown in light gray. 1010 b illustrates the centerline of the left eye FOV for the second user. 1011 b illustrates thecenter line of the right eye FOV for the second user. Note that thecenter line of the left eye FOV 1010 b for the second user and thecenter line of the right eye FOV 1011 b for the second user meet at aconvergence point 1012. This allows the second user to view a smallobject with greater detail. Note that the second user is looking in theforward direction. It is suggested that during filming of a moving thatmost of the action in the scene occur in this forward looking direction.

FIG. 10E illustrates a third user's personalized field of view (FOV) ata given time point. 1000 illustrates the left camera. 1001 illustratesthe right camera. 1006 c illustrates the left boundary of the left eyeFOV for the third user, which is shown in light gray. 1007 c illustratesthe right side boundary of the left eye FOV for the third user, which isshown in light gray. 1008 c illustrates the left boundary of the righteye FOV for the third user, which is shown in light gray. 1009 cillustrates the right side boundary of the right eye FOV for the thirduser, which is shown in light gray. 1010 c illustrates the center lineof the left eye FOV for the third user. 1011 c illustrates the centerline of the right eye FOV for the third user. Note that the center lineof the left eye FOV 1010 c for the third user and the center line of theright eye FOV 1011 c for the third user are approximately parallel,which is equivalent to looking at a very far distance. Note that thethird user is looking in a moderately leftward direction. Note that theoverlap of the left eye FOV and right eye FOV provide stereoscopicviewing to the third viewer.

FIG. 10F illustrates a fourth user's personalized field of view (FOV) ata given time point. 1000 illustrates the left camera. 1001 illustratesthe right camera. 1006 d illustrates the left boundary of the left eyeFOV for the fourth user, which is shown in light gray. 1107 dillustrates the right side boundary of the left eye FOV for the fourthuser, which is shown in light gray. 1008 d illustrates the left boundaryof the right eye FOV for the fourth user, which is shown in light gray.1009 d illustrates the right side boundary of the right eye FOV for thefourth user, which is shown in light gray. 1010 d illustrates the centerline of the left eye FOV for the fourth user. 1011 d illustrates thecenter line of the right eye FOV for the fourth user. Note that thecenter line of the left eye FOV 1010 d for the fourth user and thecenter line of the right eye FOV 1011 d for the fourth user areapproximately parallel, which is equivalent to looking at a very fardistance. Note that the fourth user is looking in a far leftwarddirection. Note that the first user, second user, third user and fourthuser are all seeing different views of the movie at the same time point.It should be noted that some of the designs, such as the camera clusteror ball system as described in

FIG. 11A illustrates a top down view of the first user's left eye viewat time point 1. 1100 illustrates the left eye view point. 1101illustrates the right eye viewpoint. 1102 illustrates the portion of thefield of view (FOV) not covered by either camera. 1103 illustrates theportion of the FOV that is covered by at least one camera. 1104Aillustrates a medial portion of a high resolution FOV used by a user,which corresponds to α=+25°. This is discussed in more detail in U.S.patent application Ser. No. 17/225,610, which is incorporated byreference in its entirety.

1105A illustrates a lateral portion of a high resolution FOV used by auser, which corresponds to α=−25°.

FIG. 11B illustrates a top down view of the first user's left eye viewwherein a convergence point in close proximity to the left eye and righteye. 1100 illustrates the left eye view point.

1101 illustrates the right eye viewpoint. 1102 illustrates the portionof the field of view (FOV) not covered by either camera. 1103illustrates the portion of the FOV that is covered by at least onecamera. 1104B illustrates a medial portion of a high resolution FOV usedby a user, which corresponds to α=−5°. 1105B illustrates a lateralportion of a high resolution FOV used by a user, which corresponds toα=+45°.

FIG. 11C illustrates a left eye view at time point 1 withoutconvergence. Note that a flower 1106 is shown in the image, which islocate along the viewing angle α=0°.

FIG. 11D illustrates a left eye view at time point 2 with convergence.Note that a flower 1106 is shown in the image, which is still locatedalong the viewing angle α=0°. However, the user has converged duringthis time point. This act of convergence causes the left eye field ofview to be altered from a horizontal field of view with a rangingbetween −25° and 25° (as shown in FIGS. 11A and 11C) to a rangingbetween −5° and +45° (as shown in FIGS. 11B and 11D). This systemimproves upon the prior art because it provides stereoscopic convergenceon stereoscopic cameras by shifting the images according to the left(and right) field of views. In some embodiments, a portion of thedisplay is non-optimized, which is described in U.S. Pat. No.10,712,837, which is incorporated by reference in its entirety.

FIG. 12 illustrates the reconstruction of various stereoscopic imagesfrom previously acquired wide angle stereo images. 1200 illustratesacquiring imagery from a stereoscopic camera system. This is camerasystem is discussed in more detail in U.S. patent application Ser. No.17/225,610, which is incorporated by reference in its entirety. 1201illustrates wherein a first camera for a left eye viewing perspectiveand a second camera for a right eye viewing perspective is utilized.1202 illustrates selecting the field of view of the first camera basedon the left eye look angle and the field of view for the second camerabased on the right eye look angle. In the preferred embodiment, theselection would be performed by a computer (e.g., integrated into a headdisplay unit) based on an eye tracking system tracking eye movements ofa user. It should also be noted that in the preferred embodiment, therewould also be an image shift inward on the display closer to the noseduring convergence, which is taught in U.S. Pat. No. 10,712,837especially FIGS. 15A, 15B, 16A, and 16B, which is incorporated byreference in its entirety. 1203 illustrates presenting the left eyefield of view to a left eye of a user and the right eye field of view toa right eye of a user. There are a variety of options at this juncture.First, use composite stereoscopic image pair wherein left eye image isgenerated from at least two lenses (e.g., first optimized for close upimage and second optimized for far away imaging) and wherein right eyeimage is generated from at least two lenses (e.g., first optimized forclose up image and second optimized for far away imaging). When user islooking at nearby object, present stereoscopic image pair with nearbyobject in focus and distant objects out of focus. When user is lookingat distant object, present stereoscopic image pair with nearby objectout of focus and distant object in focus. Second, use a variety ofdisplay devices (e.g., Augmented Reality, Virtual Reality, Mixed Realitydisplays).

FIG. 13A illustrates a top down view of a home theater. 1300 illustratesthe user. 1301 illustrates the projector. 1302 illustrates the screen.Note that this immersive home theater is displays a field of view largerthan a user's 1300 field of view. For example, if a user 1300 waslooking straight forward, the home theater would display a horizontalFOV of greater than 180 degrees. Thus, the home theater's FOV wouldcompletely cover the user's horizontal FOV. Similarly, if the user waslooking straight forward, the home theater would display a vertical FOVof greater than 120 degrees. Thus, the home theater's FOV wouldcompletely cover the user's vertical FOV. An AR/VR/MR headset could beused in conjunction with this system, but would not be required. Cheapanaglyph or disposable color glasses could also be used. A conventionalIMAX polarized projector could be utilized with IMAX-type polarizeddisposable glasses. The size of the home theater could vary. The hometheater walls could be built with white, reflective panels and framing.The projector would have multiple heads to cover the larger field ofview.

FIG. 13B illustrates a side view of the home theater as shown in FIG.13A. 1300 illustrates the user. 1301 illustrates the projector. 1302illustrates the screen. Note that this immersive home theater isdisplays a field of view larger than a user's 1300 field of view. Forexample, if a user 100 was looking forward while on a recliner, the hometheater would display a vertical FOV of greater than 120 degrees. Thus,the home theater's FOV would completely cover the user's FOV. Similarly,if the user was looking straight forward, the home theater would displaya horizontal FOV of greater than 120 degrees. Thus, the home theater'sFOV would completely cover the user's FOV.

FIG. 14A illustrates a top down view of a home theater. 1400Aillustrates a first user. 1400B illustrates a first user. 1401illustrates the projector. 1402 illustrates the screen. Note that thisimmersive home theater is displays a field of view larger than the FOVof the first user 1400A or the second user 1400B. For example, if thefirst user 1400A was looking straight forward, the first user 1400Awould see a horizontal FOV of greater than 180 degrees. Thus, the hometheater's FOV would completely cover the user's horizontal FOV.Similarly, if the first user 1400A was looking straight forward, thehome theater would display a vertical FOV of greater than 120 degrees,as shown in FIG. 14B. Thus, the home theater's FOV would completelycover the user's vertical FOV. An AR/VR/MR headset could be used inconjunction with this system, but would not be required. Cheap anaglyphor polarized glasses could also be used. A conventional IMAX polarizedprojector could be utilized with IMAX-type polarized disposable glasses.The size of the home theater could vary. The home theater walls could bebuilt with white, reflective panels and framing. The projector wouldhave multiple heads to cover the larger field of view.

FIG. 14B illustrates a side view of the home theater as shown in FIG.14A. 1400A illustrates the first user. 1401 illustrates the projector.1402 illustrates the screen. Note that this immersive home theater isdisplays a field of view larger than the first user's 1400A field ofview. For example, if the first user 1400A was looking forward while ona recliner, the user would see a vertical FOV of greater than 120degrees. Thus, the home theater's FOV would completely cover the FOV ofthe first user 1400A. Similarly, if the first user 1400A was lookingstraight forward, the home theater would display a horizontal FOV ofgreater than 120 degrees. Thus, the home theater's FOV would completelycover the FOV of the first user 1400A.

A typical high resolution display has 4000 pixels over a 1.37 mdistance. This would be equivalent to 10×10⁶ pixels per 1.87 m².Consider the data for a hemisphere theater. Assume that the hemispheretheater has a radius of 2 meters. The surface area of a hemisphere is2×π×r², which s equal to (4)(3.14)(2²) or 50.24 m². Assuming that aspatial resolution was desired to be equal to that of a typical highresolution display, this would equal (50.24 m²)(10×10⁶ pixels per 1.87m²) or 429 million pixels. Assuming the frame rate of 60 frames persecond. This is 26 times the amount of data as compared to a standard 4Kmonitor.

Some embodiments, comprise constructing a home theater to match thegeometry of the projector. The preferred embodiment is sub-spherical(e.g., hemispherical). A low cost construction would be the use of areflective surfaces stitched together with a multi-head projector. Insome embodiments, a field of view comprises a spherical coverage with a4π steradians. This can be accomplished via a HDU. In some embodiments,a field of view comprises sub-spherical coverage with at least 3πsteradians. In some embodiments, a field of view comprises sub-sphericalcoverage with at least 2π steradians. In some embodiments, a field ofview comprises sub-spherical coverage with at least 1π steradians. Insome embodiments, a field of view comprises sub-spherical coverage withat least 0.5π steradians. In some embodiments, a field of view comprisessub-spherical coverage with at least 0.25π steradians. In someembodiments, a field of view comprises sub-spherical coverage with atleast 0.05π steradians. In some embodiments, a sub-spherical IMAX systemis created for an improved movie theater experience with many viewers.The chairs would be positioned in a similar position as standard movietheaters, but the screen would be sub-spherical. In some embodiments,non-spherical shapes could also be used.

FIG. 15A illustrates time point #1 wherein a user looking straight aheadand sees a horizontal field of view of approximately 60 degreeshorizontal and 40 degrees vertical with a reasonably precise field ofview (e.g., user can see shapes and colors in peripheral FOV).

FIG. 15B shows the center portion of the TV and the field of view beingobserved by the user at time point #1. Note that in some embodiments,data would be streamed (e.g., via the internet). Note that a novelfeature of this patent is called “viewing-parameter directed streaming”.In this embodiment, a viewing parameter is used to direct the datastreamed. For example, if the user 1500 were looking straight forward,then a first set of data would be streamed to correspond with thestraight forward viewing angle of the user 1500. If, however, the userwere looking at to the side of the screen, a second set of data would bestreamed to correspond with the looking to the side viewing angle of theuser 1500. Other viewing parameters that could control viewing anglesinclude, but are not limited to, the following: user's vergence; user'shead position; user's head orientation. In a broad sense, any feature(age, gender, preference) or action of a user (viewing angle, positions,etc.) could be used to direct streaming. Note that another novel featureis the streaming of at least two image qualities. For example, a firstimage quality (e.g., high quality) would be streamed in accordance witha first parameter (e.g., within user's 30° horizontal FOV and 30°vertical FOV). And, a second image quality (e.g., lower quality) wouldbe also be streamed that did not meet this criteria (e.g., not withinuser's 30° horizontal FOV and 30° vertical FOV). Surround sound would beimplemented in this system.

FIG. 15C illustrates time point #2 wherein a user looking to the user'sleft side of the screen and sees a horizontal field of view ofapproximately 60 degrees horizontal and 40 degrees vertical with areasonably precise field of view (e.g., user can see shapes and colorsin peripheral FOV).

FIG. 15D illustrates time point #2 with the field of view being observedby the user at time point #2, which is different as compared to FIG.15B. The area of interest is half that of time point #1. In someembodiments, greater detail and higher resolution of objects within asmall FOV within the scene is provided to the user. Outside of this highresolution field of view zone, a lower resolution image quality could bepresented on the screen.

FIG. 15E illustrates time point #3 wherein a user looking to the user'sright side of the screen.

FIG. 15F illustrates time point #3 and sees a circularly shapedhigh-resolution FOV.

FIG. 16A illustrates an un-zoomed image. 1600 illustrates the image.1601A illustrates a box illustrated to denote the area within image 1600that is set to be zoomed in on.

FIG. 16B illustrates a digital-type zooming in on a portion of an image.This can be accomplished via methods described in U.S. Pat. No.8,384,771 (e.g., 1 pixel turns into 4), which is incorporated byreference in its entirety. Note that a the area to be zoomed in on canbe accomplished through a variety of user inputs including: gesturetracking systems; eye tracking systems; and, graphical user interfaces(GUIs). Note that the area within the image 1601A that was of denoted inFIG. 16A is now zoomed in on as shown in 1601B. Note that the resolutionof region 1601B is equal to that of image 1600, but just larger. Notethat 1600B illustrates portions of 1600A, which are not enlarged. Notethat 1601A is now enlarged and note that portions of 1600A are notvisualized.

FIG. 17A illustrates an un-zoomed image. 1700 illustrates the image.1701A illustrates a box illustrated to denote the area within image 1700that is set to be zoomed in on.

FIG. 17B illustrates the optical-type zooming in on a portion of animage. Note that a the area to be zoomed in on can be accomplishedthrough a variety of user inputs including: gesture tracking systems;eye tracking systems; and, graphical user interfaces (GUIs). Note thatthe area within the image 1701A that was of denoted in FIG. 17A is nowzoomed in on as shown in 1701B and also note that the image inside of1701B appears higher image quality. This can be done by selectivelydisplaying the maximum quality imagery in region 1701B and enlargingregion 1701B. Not only is the cloud bigger, the resolution of the cloudis also better. Note that 1700B illustrates portions of 1700A, which arenot enlarged (Note that some of the portions of 1700A, which are notenlarged are now covered by the zoomed in region.)

FIG. 18A illustrates a single resolution image. 1800A illustrates theimage. 1801A illustrates a box illustrated to denote the area withinimage 1800A that is set to have resolution improved in.

FIG. 18B illustrates a multi-resolution image. Note that the area whereresolution is improved can be accomplished through a variety of userinputs including: gesture tracking systems; eye tracking systems; and,graphical user interfaces (GUIs) to include a joystick or controller.Note that the area within the image 1801A that was of denoted in FIG.18A is now displayed with higher resolution as shown in 1801B. In someembodiments, the image inside of 1801B can be changed in other optionsas well (e.g., different color scheme, different brightness settings,etc.). This can be done by selectively displaying a higher (e.g.,maximum) quality imagery in region 1801B without enlarging region 1701B.

FIG. 19A illustrates a large field of view wherein a first user islooking at a first portion of the image and a second user is looking ata second portion of the image. 1900A is the large field of view, whichis of a first resolution. 1900B is the location where a first user islooking which is set to become high resolution, as shown in FIG. 19B.1900C is the location where a second user is looking which is set tobecome high resolution, as shown in FIG. 19B.

FIG. 19B illustrates that only the first portion of the image in FIG.19A and that the second portion of the image in FIG. 19A are highresolution and the remainder of the image is low resolution. 1900A isthe large field of view, which is of a first resolution (lowresolution). 1900B is the location of the high resolution zone of afirst user, which is of a second resolution (high resolution in thisexample). 1900C is the location of the high resolution zone of a seconduser, which is of a second resolution (high resolution in this example).Thus, a first high resolution zone be used for a first user. And, asecond high resolution zone can be used for a second user. This systemcould be useful for the home theater display as shown in FIGS. 14A and14B.

FIG. 20A illustrates a low resolution image.

FIG. 20B illustrates a high resolution image.

FIG. 20C illustrates a composite image. Note that this composite imagehas a first portion 2000 that is of low resolution and a second portion2001 that is of high resolution. This was described in U.S. patent Ser.No. 16/893,291, which is incorporated by reference in its entirety. Thefirst portion is determined by the user's viewing parameter (e.g.,viewing angle). A point of novelty is near-real time streaming of thefirst portion 2000 with the first image quality and the second portionwith the second image quality. Note that the first portion could bedisplayed differently from the second portion. For example, the firstportion and second portion could differ in visual presentationparameters including: brightness; color scheme; or other. Thus, in someembodiments, a first portion of the image can be compressed and a secondportion of the image is not compressed. In other embodiments, acomposite image is generated with the arranging of some high resolutionimages and some low resolution images stitched together for display to auser. In some embodiments, some portions of a large (e.g., 429 millionpixel) image are high resolution and some portions of the large imageare low resolution. The portions of the large image that are highresolution will be streamed in accordance with the user's viewingparameters (e.g., convergence point, viewing angle, head angle, etc.).

FIG. 21 illustrates a method and a process for performing near-real-timestreaming of customized images.

With respect to the display 2100, the displays include, but are notlimited to the following: a large TV; an extended reality (e.g.,Augmented Reality, Virtual Reality, or Mixed Reality display); aprojector system on a screen; a computer monitor, or the like. A keycomponent of the display is the ability to track where in the image auser is looking and what the viewing parameters are.

With respect to the viewing parameters 2101, the viewing parametersinclude, but are not limited to the following: viewing angle;vergence/convergence; user preferences (e.g., objects of particularinterest, filtering—some objects rated “R” can be filtered for aparticular user, etc.).

With respect to the cloud 1202, each frame in the movie or video wouldbe of extremely large data (especially if the home theater shown inFIGS. 14A and 14B is used in combination with the camera cluster asdescribed in U.S. patent application Ser. No. 17/225,610, which isincorporated by reference in its entirety. Note that the cloud refers tostorage, databases, etc. Note that the cloud is capable of cloudcomputing. A point of novelty in this patent is the sending of theviewing parameters of user(s) to the cloud, processing of the viewingparameters in the cloud (e.g., selecting a field of view or compositestereoscopic image pair as discussed in FIG. 12 ) and determining whichportions of extremely large data to stream to optimize the individualuser's experience. For example, multiple users could have their moviesynchronized. Each would stream 2103 from the cloud their individuallyoptimized data for that particular time point onto their mobile device.And, each would then view their individually optimized data on theirdevice. This would result in an improved immersive viewing experience.For example, suppose at a single time point, there was a dinner scenewith a chandelier, a dog, an old man, a book case, a long table, acarpet and wall art. A user named Dave could be looking at the dog andDave's images would be optimized (e.g., images with maximum resolutionand optimized color of the dog are streamed to Dave's mobile device anddisplayed on Dave's HDU). A user named Kathy could be looking at thechandelier and Kathy's images would be optimized (e.g., images withmaximum resolution and optimized color of the chandelier are streamed toKathy's mobile device and displayed on Kathy's HDU). Finally, a usernamed Bob could be looking at the old man and Bob's images would beoptimized (e.g., images with maximum resolution and optimized color ofthe old man are streamed to Bob's mobile device and displayed on Bob'sHDU). It should be noted that the cloud would stored a tremendousdataset at each time point, but only portions of it would be streamedand those portions are determined by the user's viewing parametersand/or preferences. So, the book case, long table, carpet and wall artmay all be within the field of view for Dave, Kathy and Bob, but theseobjects would not be optimized for display (e.g., the highest possibleresolution of these images stored in the cloud was not streamed).

Finally, the concept of pre-emptive is introduced. If it is predictedthat an upcoming scene is may cause a specific user viewing parameter tochange (e.g., user head turn), then pre-emptively streaming of thatadditional image frames can be performed. For example, if the time of amovie is at 1:43:05 and a dinosaur is going to make a noise and pop outfrom the left side of the screen at 1:43:30. Thus, the whole scene couldbe downloaded in a low resolution format and additional sets of data ofselective portions of the FOV could be downloaded as needed (e.g., basedon user's viewing parameter, based on upcoming dinosaur scene where useris predicted to look). Thus, the dinosaur popping out will always be inits maximum resolution. Such technique creates a more immersive andimproved viewing experience.

FIG. 22A illustrates using resection in conjunction with stereoscopiccameras. Camera #1 has a known location (e.g., latitude and longitudefrom a GPS). From Camera #1, a range (2 miles) and direction (330° NorthNorth West) to an object 2200 is known. The location of the object 2200can be computed. Camera #2 has an unknown location, but the range (1mile) and direction (30° North Northeast) to the object 2200 is known.Since the object 2200's location can be computed, the geometry can besolved and the location of camera #2 determined.

FIG. 22A illustrates using resection in conjunction with stereoscopiccameras. Camera #1 has a known location (e.g., latitude and longitudefrom a GPS). Camera #1 and Camera #2 have known locations. From Camera#1, a direction (330° North Northwest) to an object 2200B is known. FromCamera #2, a direction (30° North Northeast) to an object 2200B isknown. The location of the object 2200B can be computed.

FIG. 23A illustrates a top down view of a person looking forward to thecenter of the screen of the home theater. The person 2300 is lookingforward toward the center section 2302B of the screen 2301 of the hometheater. During this time point, the streaming is customized to have thecenter section 2302B optimized (e.g., highest possible resolution), theleft section 2302A non-optimized (e.g., low resolution or black), andthe right section 2302C non-optimized (e.g., low resolution or black).Note that a monitoring system (to detect user's viewing direction andother viewing parameters, such as gesture or facial expression) or acontroller (to receive commands from the user must also be in place) tobe inputted for the appropriate streaming.

FIG. 23B illustrates a top down view of a person looking forward to theright side of the screen of the home theater. The person 2300 is lookingtoward the right side of section 2302C of the screen 2301 of the hometheater. During this time point, the streaming is customized to have theright section 2302C optimized (e.g., highest possible resolution), theleft section 2302A non-optimized (e.g., low resolution or black), andthe center section 2302B non-optimized (e.g., low resolution or black).Note that a monitoring system (to detect user's viewing direction andother viewing parameters, such as gesture or facial expression) or acontroller (to receive commands from the user must also be in place) tobe inputted for the appropriate streaming.

FIG. 24 illustrates a method, system and apparatus for optimizingstereoscopic camera settings during image acquisition during movement.2400 illustrates determining a distance of an object (e.g., use laserrange finder) at a time point. An object tracking/target tracking systemcan be implemented. 2401 illustrates adjusting a zoom setting of astereoscopic camera system to be optimized for said distance asdetermined in step 2400. In the preferred embodiment, this would beperformed when using a zoom lens, as opposed to performing digitalzooming. 2402 illustrates adjusting the distance of separation (stereodistance) between stereoscopic cameras to be optimized for said distanceas determined in step 2400. Note that there is also an option to alsoadjust the orientation of the cameras to be optimized for said distanceas determined in step 2400. 2403 illustrates acquiring stereoscopicimagery of the target at time point in step 2400. 2404 illustratesrecording, view and/or analyzing the acquired stereoscopic imagery.

What is claimed is:
 1. A method comprising: uploading via an internet auser's viewing parameter to a cloud wherein said cloud stores imagery,wherein said imagery is obtained from a camera system, wherein saidcamera system has an orientation, wherein said camera system is notlocated on a head display unit worn by said user, wherein said cloud iscapable of cloud computing, and wherein said user's viewing parametercomprises a viewing angle, wherein said viewing angle is determinedbased on said user's use of said head display unit, wherein said headdisplay unit has an orientation, and wherein said head display unit'sorientation is different from said camera system's orientation; in saidcloud, optimizing user-specific display imagery from said imagerywherein said user-specific display imagery is based on at least saidviewing parameter, wherein said user-specific display imagery comprisesa first portion and a second portion, wherein said first portion of saiduser-specific display imagery is different from said second portion ofsaid user-specific display imagery, wherein said first portion of saiduser-specific display imagery comprises a first image quality, whereinsaid first portion of said user-specific display imagery corresponds tosaid viewing angle, wherein said second portion of said user-specificdisplay imagery does not correspond to said viewing angle, wherein saidsecond portion of said user-specific display imagery comprises a secondimage quality, and wherein said second image quality is lower than saidfirst image quality; downloading via the internet said user-specificdisplay imagery; and displaying said user-specific display imagery tosaid user.
 2. The method of claim 1 further comprises: wherein saiduser-specific display imagery comprises a first portion with a firstspatial resolution and a second portion with a second spatialresolution, and wherein said first spatial resolution is higher thansaid second spatial resolution.
 3. The method of claim 1 furthercomprises: wherein said imagery comprises video imagery, and whereineach frame of said video imagery is divided into said first portion andsaid second portion.
 4. The method of claim 1 further comprises: whereinsaid user-specific display imagery comprises wherein said first portioncomprises a first zoom setting and wherein said second portion comprisesa second zoom setting, and wherein said first zoom setting is higherthan said second zoom setting.
 5. The method of claim 4 furthercomprises wherein said first portion is determined by at least one ofthe group consisting of: a position of said user's body; an orientationof said user's body; a gesture of said user's hand; a facial expressionof said user; a position of said user's head; and an orientation of saiduser's head.
 6. The method of claim 4 further comprises wherein saidfirst portion is determined by a graphical user interface.
 7. The methodof claim 1 further comprising: wherein said imagery comprises a firstfield of view (FOV), wherein said user-specific display imagerycomprises a second FOV, and wherein said first FOV is larger than saidsecond FOV.
 8. The method of claim 1 further comprising: wherein saidimagery comprises stereoscopic imagery; and wherein said stereoscopicimagery is obtained via stereoscopic cameras or stereoscopic cameraclusters.
 9. The method of claim 1 further comprising wherein saidimagery comprises stitched imagery wherein said stitched imagery isgenerated by at least two cameras.
 10. The method of claim 1 furthercomprising: wherein said imagery comprises composite imagery, whereinsaid composite imagery is generated by: taking a first image of a scenewith a first set of camera settings wherein said first set of camerasettings causes a first object to be in focus and a second object to beout of focus; and taking a second image of a scene with a second set ofcamera settings wherein said second set of camera settings causes saidsecond object to be in focus and said first object to be out of focus.11. The method of claim 10 further comprises wherein: when said userlooks at said first object, said first image would be presented to saiduser; and when said user looks at said second object, said second imagewould be presented to said user.
 12. The method of claim 10 furthercomprising combining at least said first object from said first imageand said second object from said second image into said composite image.13. The method of claim 1 further comprises wherein said viewing angleis movable by said user.
 14. The method of claim 1 further compriseswherein said viewing parameter comprises convergence.
 15. The method ofclaim 1 further comprising: wherein said user-specific display imageryis 3D imagery, and wherein said 3D imagery is presented on said headdisplay unit (HDU).
 16. The method of claim 15 further comprisingwherein said viewing angle is determined by an orientation of said HDU.17. A method comprising: determining a user's viewing parameter whereinsaid user's viewing parameter comprises a viewing angle, wherein saidviewing angle is determined based on said user's head orientation, andwherein a head display unit (HDU) determines said user's headorientation; sending via an internet said user's viewing parameter to acloud wherein said cloud is capable of cloud computing, wherein saidimagery is obtained from a camera system, wherein said camera system hasan orientation, wherein said head display unit's orientation isdifferent from said camera system's orientation, wherein said camerasystem is not located on said HDU worn by said user, wherein said cloudcomputing generates user-specific display imagery from imagery stored onsaid cloud, wherein said user-specific display imagery is based on atleast said user's viewing parameter, wherein said user-specific displayimagery comprises a first portion and a second portion, wherein saidfirst portion of said user-specific display imagery is different fromsaid second portion of said user-specific display imagery, wherein saidfirst portion of said user-specific display imagery comprises a firstimage quality, wherein said first portion of said user-specific displayimagery corresponds to said viewing angle, wherein said second portionof said user-specific display imagery does not correspond to saidviewing angle, wherein said second portion of said user-specific displayimagery comprises a second image quality, and wherein said second imagequality is lower than said first image quality; receiving via saidinternet said user-specific display imagery; and displaying saiduser-specific display imagery on said HDU wherein said HDU comprises aleft eye display and a right eye display.
 18. The method of claim 1further comprising wherein said user-specific display imagery ispresented to said user on a display wherein said user has at least a0.5π steradian field of view.
 19. The method of claim 17 furthercomprising wherein said camera system comprises a stereoscopic camerasystem.
 20. A method comprising: receiving via an internet a user'sviewing parameter at a cloud wherein said user's viewing parametercomprises a viewing angle, wherein said viewing angle is determinedbased on said user's look angle, wherein an eye tracking system of ahead display unit (HDU) determines an orientation of said user's lookangle, and wherein said cloud is capable of cloud computing, using cloudcomputing to generate user-specific display imagery from imagery storedon said cloud, wherein said imagery is obtained from a camera system,wherein said camera system has an orientation, wherein said orientationof said user's look angle is different from said camera system'sorientation, wherein said camera system is not located on said HDU wornby said user, wherein said user-specific display imagery is based on atleast said user's viewing parameter, wherein said user-specific displayimagery comprises a first portion and a second portion, wherein saidfirst portion of said user-specific display imagery is different fromsaid second portion of said user-specific display imagery, wherein saidfirst portion of said user-specific display imagery comprises a firstimage quality, wherein said first portion of said user-specific displayimagery corresponds to said viewing angle, wherein said second portionof said user-specific display imagery does not correspond to saidviewing angle, wherein said second portion of said user-specific displayimagery comprises a second image quality, and wherein said second imagequality is lower than said first image quality; sending via saidinternet said user-specific display imagery to said HDU wherein said HDUcomprises a left eye display and a right eye display, wherein said HDUdisplays said user-specific display imagery.